Papers with contrastive fine-tuning
Multi-Tenant Optimization For Few-Shot Task-Oriented FAQ Retrieval (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Business-specific Frequently Asked Questions (FAQs) are important part of many taskoriented dialog systems today. |
| Approach: | They propose a method to scale multi-tenant FAQ applications in real-world context . they use query-Question similarity and few-shot intent detection techniques . |
| Outcome: | The proposed approach scales multi-tenant FAQ applications in real-world context . it uses fine-tuning of the last layer in sentence Bi-Encoders along with tenant-specific weight switching. |
Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval (2024.acl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that Transformer-based language models lose information in the middle of input sequences, especially in the context of web document retrieval. |
| Approach: | They examine position biases at multiple stages of the training pipeline for an encoder-decoder neural retrieval model, namely language model pre-training, contrastive pre- training, and contrastive fine-tuning. |
| Outcome: | The proposed model generates embeddings that better capture the beginning of the input content, with fine-tuning further aggravating this effect. |
Contrastive Fine-tuning Improves Robustness for Neural Rankers (2021.findings-acl)
Copied to clipboard
| Challenge: | Current state-of-the-art neural rankers can deteriorate when exposed to noisy inputs or applied to a new domain. |
| Approach: | They propose a contrastive loss and ranking loss method for fine-tuning rankers that combine ranking loss and rank loss to improve their robustness to query reformulations and noise perturbations. |
| Outcome: | The proposed method outperforms data augmentation for robustifying rankers on four passage ranking datasets. |
Improving LLM Domain Certification with Pretrained Guide Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate off-domain or harmful responses when deployed in high-stakes domains. |
| Approach: | They propose a method that leverages pretrained language models as guide models to sharply distinguish acceptable from refused content. |
| Outcome: | The proposed approach exploits pretrained language models as guide models while aligned to the target domain. |
RECIPE-TKG: From Sparse History to Structured Reasoning for LLM-based Temporal Knowledge Graph Completion (2026.eacl-long)
Copied to clipboard
| Challenge: | Temporal Knowledge Graphs (TKGs) represent dynamic facts as timestamped relations between entities. Large Language Models (LLMs) have sparked interest in using pretrained generative models for TKG completion. |
| Approach: | They propose a framework that allows for rule-based multi-hop sampling and contrastive fine-tuning to shape relational compatibility. |
| Outcome: | Experiments show that RECIPE-TKG outperforms prior LLM-based methods across input regimes. |
A Two-Stage Prediction-Aware Contrastive Learning Framework for Multi-Intent NLU (2024.lrec-main)
Copied to clipboard
| Challenge: | Multi-intent natural language understanding (NLU) models lack the rich information between the shared intents, especially in low-data scenarios. |
| Approach: | They propose a two-stage framework for multi-intent natural language understanding to harness shared intent information by word-level pre-training and prediction-aware contrastive fine-tuning. |
| Outcome: | The proposed framework surpasses baselines on low-data and full-data scenarios. |
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2022.naacl-main)
Copied to clipboard
Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev
| Challenge: | Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. |
| Approach: | They propose a typology of factual errors to better understand hallucinations generated by current models and a contrastive fine-tuning strategy to improve the factual consistency and overall quality of summaries. |
| Outcome: | The proposed model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarizing datasets. |
Contrastive Training Improves Zero-Shot Classification of Semi-structured Documents (2023.findings-acl)
Copied to clipboard
| Challenge: | Xu et al., 2020 focus on semi-structured document classification in a zero-shot setting . positional, layout, and style information play a vital role in interpreting such documents . |
| Approach: | They propose a matching-based approach that relies on a pairwise contrastive objective for pretraining and fine-tuning. |
| Outcome: | The proposed method significantly improves Macro F1 in the zero-shot learning setting. |
Multi-Label Intent Detection via Contrastive Task Specialization of Sentence Encoders (2022.emnlp-main)
Copied to clipboard
Ivan Vulić, Iñigo Casanueva, Georgios Spithourakis, Avishek Mondal, Tsung-Hsien Wen, Paweł Budzianowski
| Challenge: | a novel framework for task-oriented dialog ToD systems is proposed . a task of recognizing the user's intent or goal from their utterance is a crucial component of any TOD system. |
| Approach: | They propose to transform general-purpose sentences into task-specialized SEs by contrastive fine-tuning on annotated multi-label data. |
| Outcome: | The proposed framework yields effective mID models with large gains over non-specialized models across a spectrum of different m ID datasets. |
Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Embedding a sentence into a point in a highdimensional continuous space plays a foundational role in the natural language processing. |
| Approach: | They propose to use contrastive loss to fine-tune sentences by inverse word frequency . they also show that more informative words receive greater weight than less informative ones . |
| Outcome: | The proposed method improves the performance of sentence embeddings by weighing them based on information-theoretic quantities. |